Streaming Multiscale Deep Equilibrium Models

AbstractWe present StreamDEQ, a method that infers frame-wise representations on videos with minimal per-frame computation. In contrast to conventional methods where compute time grows at least linearly the network depth, we aim update in continuous manner. For this purpose, leverage recently emerging implicit layer models, which infer representation of an image by solving fixed-point problem. Our main insight is slowly changing nature and use previous frame as initial condition each frame. This scheme effectively recycles recent inference computations greatly reduces needed processing time. Through extensive experimental analysis, show StreamDEQ able recover near-optimal few frames time, maintain up-to-date throughout video duration. experiments semantic segmentation object detection achieves par accuracy baseline (standard MDEQ) while being more than \(3\times \) faster. Code additional results are available https://ufukertenli.github.io/streamdeq/.KeywordsImplicit modelsVideo analysis understandingVideo detectionVideo